Image processing device and method, and computer readable recording medium containing program

ABSTRACT

A plurality of original images of a subject viewed from different viewpoints are acquired. Then, at least one interpolation image to interpolate at least one new viewpoint between the viewpoints of the original images or outside the viewpoints of the original images is generated from the original images. Then, identification information indicating whether each image is the original image or the interpolation image and viewpoint information representing the order of viewpoint of each image are added to the original images and the interpolation image, and the original images and the interpolation image with the identification information and the viewpoint information added thereto are stored in an associated fashion.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device and methodfor generating an interpolation image to interpolate a new viewpointfrom a plurality of original images viewed from different viewpoints,and a computer readable recording medium containing a program forcausing a computer to carry out the image processing method.

2. Description of the Related Art

A stereoscopic image which allows stereoscopic viewing based on adisparity can be acquired with a stereoscopic imaging device, such as astereo camera, which acquires images using at least a pair of imagingmeans placed at different positions. Such a stereoscopic imaging deviceacquires two left right images by imaging the same subject fromdifferent viewpoints with the pair of imaging means which are providedwith a convergence angle between optical axes thereof, and generates thestereoscopic image, which allows stereoscopic viewing based on adisparity, from the two images.

A technique to store the plurality of images acquired by thestereoscopic imaging device as a single image file has been proposed(see U.S. Patent Application Publication No. 20050243187). According tothis technique, these images can be managed as a single image file,thereby preventing large increase of the number of image files.

In stereoscopic imaging, a larger convergence angle between the imagingmeans for acquiring the stereoscopic images results in a largerdisparity between the acquired images. If the disparity between theimages is excessively large, a disparity between the left and rightimages in the horizontal direction on the stereoscopic display plane istoo large to provide a natural view. In this case, the disparity betweenadjacent images can be reduced by estimating a new viewpoint between thepositions of the imaging means and generating an interpolation imagewhich interpolates between the two images, thereby allowing goodstereoscopic viewing. It should be noted that the viewpoint of theinterpolation image to be generated may be estimated not only betweenthe viewpoints of the plurality of images but also outside theviewpoints of the plurality of images.

When the interpolation image is generated in this manner, however, theoriginal images which have been acquired through imaging and thegenerated interpolation image are managed as separate files. This mayoften lead to confusion between the original images and theinterpolation image, and when the interpolation image is re-created fromthe original images or only the interpolation image is deleted to reducethe file volume, the original image may erroneously be deleted.

SUMMARY OF THE INVENTION

In view of the above-described circumstances, the present invention isdirected to facilitating management of the original images and theinterpolation image.

An aspect of the image processing device according to the inventionincludes: image acquiring means for acquiring a plurality of originalimages of a subject viewed from different viewpoints; interpolationimage generating means for generating, from the original images, atleast one interpolation image to interpolate at least one new viewpointbetween the viewpoints of the original images or outside the viewpointsof the original images; and storing means for adding identificationinformation and viewpoint information to the original images and theinterpolation image, the identification information indicating whethereach image is the original image or the interpolation image and theviewpoint information representing an order of viewpoint of each image,and storing the original images and the interpolation image in anassociated fashion with the identification information and the viewpointinformation added thereto.

The description “storing the original images and the interpolation imagein an associated fashion” means that the original images and theinterpolation image are associated in an integral fashion in view ofmanagement of the images. Specifically, the original images and theinterpolation image may be stored in an associated fashion by, forexample, storing the original images and the interpolation image in thesame folder, assigning a common file name to the original images and theinterpolation image, storing the original images and the interpolationimage in a single file, or recording, in the header of at least one ofthe original images and the interpolation image, file names of the otherimages. It should be noted that, even when the original images and theinterpolation image are stored in an associated fashion, operations on,such as accessing to, one of the original images and the interpolationimage are permitted.

In the image processing device according to the invention, the storingmeans may store the original images and the interpolation image as asingle image file.

In the image processing device according to the invention, the storingmeans may store the interpolation image at an area following areas forstoring the original images in the image file.

The description “stores the interpolation image at an area followingareas for storing the original images” means, for example, that, whenthe original images and the interpolation image are recorded in anintegral fashion in successive address spaces of a recording medium, thearea storing the interpolation image may be a subordinate area to theareas storing the original images.

In the image processing device according to the invention, when aninstruction to reduce a file volume is made, the storing means maydelete only the interpolation image.

The “instruction to reduce a file volume” may include, for example, aninstruction to increase available space of a recording medium forrecording the image file of the original images and the interpolationimage, an instruction to delete the image file, or an instruction totransfer the image file, which is more easily achieved when the filevolume is smaller.

The “instruction to reduce a file volume” may be made for one of theassociated image files of the original images and interpolation image.For example, when the file volume is reduced by deletion, a deletionmenu including an instruction to delete all the original images and theinterpolation image with respect to a selected image and an instructionto reduce the volume with respect to the selected image may be provided.Then, if the latter is instructed, only the interpolation image may bedeleted. If, for example, the user has instructed to delete some of theoriginal images and the interpolation image (for example, oneinterpolation image), presence of images related to the selected imagemay be notified to the user, and the fact that the above-describeddeletion menu (or at least the instruction to reduce the volume in thedeletion menu) is prepared may be presented to the user.

In the image processing device according to the invention, when theoriginal images are acquired through imaging, the interpolation imagegenerating means may determine a number of the at least oneinterpolation image based on a positional relationship between imagingmeans used to acquire the original images.

The positional relationship between the imaging means used to acquirethe original images may include, for example, at least one of aconvergence angle and a baseline length of the plurality of imagingmeans.

An aspect of the imaging device according to the invention includes: aplurality of imaging means disposed at different positions from eachother for acquiring the original images; and the image processing deviceaccording to the invention.

An aspect of the image processing method according to the inventionincludes: acquiring a plurality of original images of a subject viewedfrom different viewpoints; generating, from the original images, atleast one interpolation image to interpolate at least one new viewpointbetween the viewpoints of the original images or outside the viewpointsof the original images; and adding identification information andviewpoint information to the original images and the interpolationimage, the identification information indicating whether each image isthe original image or the interpolation image and the viewpointinformation representing an order of viewpoint of each image, andstoring the original images and the interpolation image in an associatedfashion with the identification information and the viewpointinformation added thereto.

It should be noted that the image processing method according to theinvention may also be provided in the form of a computer readablerecording medium containing a program for causing a computer to carryout the image processing method.

EFFECT OF THE INVENTION

According to the invention, the identification information indicatingwhether each image is the original image or the interpolation image andthe viewpoint information representing the order of viewpoint are addedto each of the original images and the at least one interpolation image,and the original images and the interpolation image with theidentification information and the viewpoint information added theretoare stored in an associated fashion. This facilitates management of theoriginal images and the interpolation image. In particular, since theidentification information indicating whether each image is the originalimage or the interpolation image is added to each of the original imagesand the interpolation image, such a situation that the original image iserroneously deleted can be prevented. Further, addition of the viewpointinformation representing the order of viewpoint facilitates generationof the stereoscopic image and re-creation of the interpolation image.

Further, by storing the original images and the interpolation image asthe single image file, the number of files is not largely increased,thereby facilitating management of the original images and theinterpolation image.

In this case, if the interpolation image is stored at an area followingareas for storing the original images in the image file, it is notnecessary to re-create the file when the interpolation image is deleted,thereby facilitating deletion of the interpolation image.

Moreover, when an instruction to reduce the file volume is made, onlythe interpolation image can be deleted, thereby facilitating reductionof the file volume.

In addition, by determining the number of the at least one interpolationimage based on the positional relationship between the imaging meansused to acquire the original images when the original images areacquired through imaging, appropriate number of interpolation images canbe generated to provide appropriate number of viewpoints.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram illustrating the configuration of astereoscopic imaging device to which an image processing deviceaccording to one embodiment of the present invention is applied,

FIG. 2 is a diagram illustrating the configuration of each imaging unit,

FIG. 3 is a diagram for explaining placement of the imaging units,

FIG. 4 illustrates an example of original images,

FIG. 5 is a diagram for explaining generation of interpolation images,

FIG. 6 illustrates a relationship between a 3D shape model and imageprojection planes on which the original images are acquired,

FIG. 7 is a diagram illustrating the file composition of an originalimage file according to one embodiment of the invention,

FIG. 8 is a diagram illustrating the file composition of an image filecontaining the original images and the interpolation images according toone embodiment of the invention,

FIG. 9 is a flow chart illustrating a process carried out when theinterpolation image is generated in one embodiment of the invention, and

FIG. 10 is a flow chart illustrating a process carried out when aninstruction to reduce the file volume is made.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an embodiment of the present invention will be describedwith reference to the drawings. FIG. 1 is a schematic block diagramillustrating the configuration of a stereoscopic imaging device to whichan image processing device according to the embodiment of the inventionis applied. As shown in FIG. 1, the stereoscopic imaging device 1according to this embodiment includes two imaging units 2A and 2B, animaging control unit 3, an image processing unit 4, an interpolationimage generating unit 5, a file generating unit 6, a stereoscopic imagegenerating unit 7, a frame buffer 8, a media control unit 9, an inputunit 10 including operation buttons for various inputs, a display unit11, such as a liquid crystal monitor, for displaying various items, anda controller 12, which are connected via a bus 13.

FIG. 2 illustrates the configuration of the imaging units 2A and 2B. Asshown in FIG. 2, each of the imaging units 2A and 2B includes a lens20A, 20B, an aperture diaphragm 21A, 21B, a shutter 22A, 22B, a CCD 23A,23B, ananalog front end (AFE) 24A, 24B and an A/D converting unit 25A,25B. It should be noted that, as shown in FIG. 3, the imaging units 2Aand 2B are placed to view a subject with a convergence angle α and apredetermined baseline length K. It should be noted that information ofthe convergence angle α and the baseline length K are stored in a ROM12C, which will be described later.

Each of the lenses 20A and 20B is formed by a plurality of lenses havingtheir respective functions, such as a focusing lens for focusing on thesubject and a zooming lens for effecting a zooming function, and theposition of each lens is adjusted by a lens driving unit (not shown)based on focused focal position data obtained through AF processingcarried out by the imaging control unit 3 and zoom data obtained throughoperation of a zoom lever (not shown).

Aperture diameters of the aperture diaphragms 21A and 21B are adjustedby a diaphragm aperture driving unit (not shown) based on aperture valuedata obtained through AE processing carried out by the imaging controlunit 3.

The shutters 22A and 22B are mechanical shutters, and are driven by ashutter driving unit (not shown) according to a shutter speed obtainedthrough the AE processing.

Each of the CCDs 23A and 23B includes a photoelectric surface, on whicha large number of light-receiving elements are arrangedtwo-dimensionally. A light image of the subject is focused on eachphotoelectric surface and is subjected to photoelectric conversion toobtain an analog imaging signal. Further, a color filter formed byregularly arrayed R, G and B color filters are disposed on the frontside of each CCD 23A, 23B.

The AFEs 24A and 24B process the analog imaging signals fed from theCCDs 23A and 23B to remove noise from the analog imaging signals andadjust gain of the analog imaging signals (this operation is hereinafterreferred to as “analog processing”).

The A/D converting units 25A and 25B convert the analog imaging signals,which have been subjected to the analog processing by the AFEs 24A and24B, into digital imaging signals. The images represented by digitalimage data acquired by the imaging units 2A and 2B are referred to asimages L1 and R1, respectively.

The imaging control unit 3 includes an AF processing unit and an AEprocessing unit (not shown). When a release button included in the inputunit 10 is half-pressed, the imaging units 2A and 2B acquire preliminaryimages, and the AF processing unit determines focal distances for thelenses 20A and 20B based on the preliminary images and outputs theinformation to the imaging units 2A and 2B. The AE processing unitdetermines an aperture value and shutter speed based on the preliminaryimages, and outputs the information to the imaging units 2A and 2B. Itshould be noted that predetermined focal position, aperture value andshutter speed may be used for imaging without carrying out the AFprocessing and the AE processing.

When the release button is fully pressed, the imaging control unit 3instructs the imaging units 2A and 2B to acquire the images L1 and R1which are actually used to generate a stereoscopic image. It should benoted that, before the release button is operated, the imaging controlunit 3 instructs the imaging units 2A and 2B to acquire live viewimages, which have fewer pixels than the actual images, for checkingimaging ranges of the imaging units 2A and 2B.

As shown in FIG. 3, the images L1 and R1 are acquired by imaging thesubject from two different imaging positions. As shown in FIG. 4, thereis a disparity between images of the subject contained in the images L1and R1 depending on a difference between the imaging positions. Itshould be noted that the images L1 and R1 are acquired at imagingpositions of the imaging units 2A and 2B, i.e., imaging positions on theleft and right facing the subject, and a stereoscopic image is generatedbased on the images L1 and R1. Therefore, in the following description,the images L1 and R1 are referred to as original images L1 and R1. Theimage L1 is displayed on the left and the image R1 is displayed on theright when the stereoscopic image is generated.

Although the stereoscopic image is generated from the two originalimages L1 and R1 in this embodiment, three or more imaging units may beprovided and more than one images may be generated from three or moreoriginal images acquired through imaging at three or more differentimaging positions.

The image processing unit 4 applies, to the digital image data of theoriginal images L1 and R1 acquired by the imaging units 2A and 2B,correction to correct for, for example, misalignment between angles ofview of the imaging units 2A and 2B, difference between zooming factors,misalignment between images due to rotation of the CCD, and trapezoidaldistortion due to the imaging units 2A and 2B imaging the subject withthe convergence angle α therebetween, and image quality correction, suchas white balance adjustment, tone correction, sharpness correction andcolor correction. In this description, the original images L1 and R1which have been processed by the image processing unit 4 are alsodesignated by the same reference symbols L1 and R1.

The interpolation image generating unit 5 generates at least oneinterpolation image from the original images L1 and R1 acquired throughimaging or the original images L1 and R1 which have been acquiredthrough imaging and recorded in a medium 9A. In this embodiment, thenumber of the interpolation images to be generated is determined dependson the magnitude of the convergence angle α of the imaging units 2A and2B.

When a stereoscopic image is generated and used for stereoscopicviewing, if the convergence angle of the imaging units 2A and 2B is aslarge as one degree or more, for example, the disparity between theoriginal images L1 and R1, that is, the disparity between left and rightimages in the horizontal direction on a stereoscopic display plane, istoo large to provide a natural view. Therefore, the interpolation imagegenerating unit 5 determines whether or not the convergence angle α ofthe imaging units 2A and 2B described in the ROM 12C or described asheader information of an image file, which will be described later, isless than one degree. If the angle is not less than one degree, theinterpolation image generating unit 5 sets, as an interpolation number N(i.e., the number of interpolation images), a maximum N value whichprovides a value obtained by dividing the convergence angle α with N+1being less than one. Namely, the maximum N value that satisfies therelationship of inequality (1) below is found and set as theinterpolation number N:1 degree<α/(N+1)  (1)Then, the interpolation images of the set interpolation number N aregenerated.

In some cases, the stereoscopic imaging device 1 according to thisembodiment may generate the interpolation image from images which havebeen acquired by an imaging device (or devices) with a differentconvergence angle from that of the imaging units 2A and 2B of thestereoscopic imaging device 1. There is no problem if the convergenceangle is described in the header of the image file of the images;however, the convergence angle may not be described. In this case, theinterpolation image generating unit 5 finds corresponding points betweenthe original images, and calculates the convergence angle based on thecorresponding points.

FIG. 5 is a diagram for explaining generation of the interpolationimage. In this example, the interpolation number is two. As shown inFIG. 5, the interpolation image generating unit 5 generates twointerpolation images H1 and H2 from the original images L1 and R1. Theinterpolation images H1 and H2 are generated such that a convergenceangle between the original image L1 and the interpolation image H1 nextto each other, a convergence angle between the interpolation image H1and the interpolation image H2 next to each other, and a convergenceangle between the interpolation image H2 and the original image R1 nextto each other are the same.

The interpolation image generating unit 5 generates the interpolationimages H1 and H2 by applying morphing to the original images L1 and R1such that the disparity between the corresponding pixels contained inthe original images L1 and R1 is gradually decreased. Specifically,corresponding points between the left and right images are detected, andthe corresponding points are connected to each other with straight linesor curved lines. Then, the straight lines or curved lines are divided tocalculate quasi-corresponding points. Then, the original images L1 andR1 are deformed to conform to the quasi-corresponding points to generatethe interpolation images H1 and H2. Alternatively, any of knowntechniques, such as one disclosed in U.S. Patent Application PublicationNo. 20030016871, may be used.

It should be noted that, although the interpolation images generated inthis example represent images viewed from viewpoints between theviewpoints from which the original images L1 and R1 are viewed, aninterpolation image H3 viewed from a viewpoint outside the originalimage L1, as shown in FIG. 5, may be generated by estimating theviewpoint outside the original image L1.

In stead of applying morphing, a 3D shape model of the subject containedin the original images L1 and R1 may be generated to generate theinterpolation images from the 3D shape model. The 3D shape model can becreated by finding corresponding points between the original images L1and R1, and calculating distances from the imaging positions to thecorresponding points on the subject according to the principle oftriangulation based on the disparity between the corresponding points.When the subject is imaged by imaging units 2A and 2B to provide theoriginal images L1 and R1, each of the original images L1 and R1 isformed by a projection image formed by a collection of intersectionpoints between straight lines connecting points on the subject with thefocal point of each imaging unit 2A, 2B and an image projection plane (aplane corresponding to the imaging surface of each of the CCDs 23A and23B of the imaging units 2A and 2B) onto which the image is projected.FIG. 6 illustrates a relationship between the 3D shape model and theimage projection planes on which the original images L1 and R1 areacquired. In this example, the 3D shape model is a cube, for the purposeof explanation. As shown in FIG. 6, the interpolation image may begenerated by setting a hypothetical image projection plane or planes Tof the number corresponding to the interpolation number (theinterpolation number is one in the example of FIG. 6) between the imageprojection planes of the original images L1 and R1, and calculating aprojection image on the hypothetical image projection plane T.

The file generating unit 6 compresses the image data of the originalimages L1 and R1 according to a certain compression format, such asJPEG, to generate an original image file F0. In a case where theinterpolation images H1 and H2 are generated, an image file F1 isgenerated from the image data of the original images L1 and R1 and theinterpolation images H1 and H2. A header storing associated information,such as imaging time and date, based, for example, on the Exif format isadded to the image file.

FIG. 7 illustrates the file composition of the original image file inthis embodiment. As shown in FIG. 7, the image file F0 includes a headerarea (header 1) 40 for the original image L1, an image data area 41 forthe original image L1, a header area (header 2) 42 for the originalimage R1, and an image data area 43 for the original image R1, which arearranged in this order.

In the header area 40 for the original image L1, address start position,attribute information and associated information are described as headerinformation. The address start position contains a start position of theaddress of the header area 42 of the original image R1, which isdescribed in the form of a list. The attribute information contains theorder of viewpoint, image identification code and imaging conditioninformation. The order of viewpoint represents the order of theviewpoints when the original images L1 and R1 are viewed from the left,and the order of viewpoint for the original image L1 and that for theoriginal image R1 are respectively “1” and “2”. The image identificationcode is information indicating whether the image stored in the imagedata area corresponding to the header area is the original image or theinterpolation image. Namely, if the image is the original image, thecode is “1”, and if the image is the interpolation image, the code is“2”. The imaging condition information contains the convergence angle αand the baseline length K of the imaging units 2A and 2B, and theinformation stored in the ROM 12C is used. Thus, for the original imageL1, the order of viewpoint is “1”, and the image identification code is“1” which indicates that the image is the original image. The associatedinformation contains the imaging time and date.

The header area 42 for the original image R1 contains, as the headerinformation, the attribute information of the original image R1, i.e.,the order of viewpoint (=2) and the image identification code (=1).

FIG. 8 illustrates the file composition of the image file containing theoriginal images and the interpolation images in this embodiment. Asshown in FIG. 8, the image file F1 includes a header area (header 1) 50for the original image L1, an image data area 51 for the original imageL1, a header area (header 2) 52 for the original image R1, an image dataarea 53 for the original image R1, a header area (header 3) 54 for theinterpolation image H1, an image data area 55 for the interpolationimage H1, a header area (header 4) 56 for the interpolation image H2 andan image data area 57 for the interpolation image H2, which are arrangedin this order.

The header area 50 for the original image L1 contains, as the headerinformation, the address start position, the attribute information andthe associated information. The address start position contains startpositions of addresses of the header areas 52, 54 and 56 of the originalimage R1 and the interpolation images H1 and H2 (i.e., the headers 2, 3and 4), which are described in the form of a list. The attributeinformation contains the order of viewpoint, the image identificationcode and the imaging condition information. The order of viewpointrepresents the order of the viewpoints when the original images L1 andR1 and the interpolation images H1 and H2 are viewed from the left, andthe order of viewpoint for the original image L1, the order of viewpointfor the interpolation image H1, the order of viewpoint for theinterpolation image H2 and the order of viewpoint for the original imageR1 are respectively “1”, “2”, “3”, and “4”. The image identificationcode is information indicating whether the image stored in the imagedata area corresponding to the header area is the original image or theinterpolation image. Namely, if the image is the original image, thecode is “1”, and if the image is the interpolation image, the code is“2”. The imaging condition information contains the convergence angle αand the baseline length K of imaging units 2A and 2B. Thus, for theoriginal image L1, the order of viewpoint is “1”, and the imageidentification code is “1” which indicates that the image is theoriginal image. The associated information contains the imaging time anddate.

The header area 52 for the original image R1 contains, as the headerinformation, the attribute information of the original image R1, i.e.,the order of viewpoint (=4) and the image identification code (=1).

The header area 54 for the interpolation image H1 contains, as theheader information, the attribute information of the interpolation imageH1, i.e., the order of viewpoint (=2), the image identification code(=2) and information of an interpolation method used. The interpolationmethod here is a technique used to generate the interpolation image, andin this case, information of the technique used by the interpolationimage generating unit 5 to generate the interpolation images H1 and H2is described.

The header area 56 for the interpolation image H2 contains, as theheader information, the attribute information of the interpolation imageH2, i.e., the order of viewpoint (=3), the image identification code(=2) and the information of the interpolation method used.

The stereoscopic image generating unit 7 applies, to the original imagesL1 and R1 and the interpolation images H1 and H2, stereoscopic displayprocessing to generate a stereoscopic image to allow stereoscopicviewing on the display unit 11. It should be noted that the conversionfor stereoscopic display can be achieved, for example, by overlappingthe images forming a pair (i.e., the adjacent images with respect to theorder of the viewpoints) with changing the colors of the images into redand blue, for example (anaglyph system), by overlapping the imagesforming a pair with converting the images to have different polarizationdirections (polarization filter system), or by combining the imagesforming a pair line by line in an alternating manner (parallax barriersystem and lenticular system), depending on the manner of stereoscopicviewing and whether the display unit 11 is a 3D liquid crystal display.

The frame buffer 8 is a storing unit for temporarily storing the imageto be displayed on the display unit 11.

The media control unit 9 accesses the recording medium 9A to controlwriting and reading of the image file F0.

The controller 12 controls the units of the device 1 and includes a CPU12A, a RAM 12B, which provides work space for various processing carriedout by the device 1, such as generating the interpolation image,generating the stereoscopic image, generating the image file anddisplaying the stereoscopic image, as will be described later, and theROM 12C, which stores operation programs for the device 1, variousconstants, etc.

Next, processes carried out in this embodiment are described. FIG. 9 isa flow chart illustrating a process carried out when the interpolationimage is generated in this embodiment. In this example, the originalimages L1 and R1 have been acquired through imaging and recorded in themedium 9A. When an instruction to generate the interpolation image isinputted via the input unit 10, the controller 12 starts the process andthe media control unit 9 reads out the original image file F0 from themedium 9A (step ST1). The original image file F0 is decompressed in theRAM12B, and the interpolation image generating unit 5 determines whetheror not the convergence angle α is described in the imaging conditioninformation contained in the header information of the original imagefile F0 (step ST2). If the determination in step ST2 is negative, theconvergence angle α is calculated from the distribution of thecorresponding points between the original images L1 and R1 (step ST3).It should be noted that, in a case where the convergence angle α isalways described in the original image file F0, the determination instep ST2 may be omitted.

If determination in step ST2 is affirmative, or after step ST3,determination is made as to whether or not the convergence angle α isless than one degree (step ST4). If the determination in step ST4 isaffirmative, it is not necessary to generate the interpolation image,and the process ends. At this time, an alarm informing that it is notnecessary to generate the interpolation image may be displayed on thedisplay unit 12. If the determination in step ST4 is negative, theinterpolation image generating unit 5 sets the interpolation number(step ST5). Setting of the interpolation number is achieved bycalculating the maximum value of N that satisfies the above describedinequality (1).

Then, the interpolation image generating unit 5 generates theinterpolation image (or images) of the interpolation number N from theoriginal images L1 and R1 (step ST6). Subsequently, the file generatingunit 6 sets the order of the viewpoints of the original images L1 and R1and the interpolation image (step ST7), and generates the addresses ofthe data areas for the original images L1 and R1 and the interpolationimage in the image file to be generated (step ST8). Then, the filegenerating unit 6 generates the header information for the interpolationimage (step ST9), updates the header information for the original imagesL1 and R1 (step ST10), and generates the image file F1 from the originalimages L1 and R1 and the interpolation image (step ST11). Then, themedia control unit 9 records the image file F1 in the medium 9A (stepST12), and the process ends.

Next, a process carried out when an instruction to reduce the filevolume of the generated image file F1 is made is described. FIG. 10 is aflow chart illustrating the process carried out when the instruction toreduce the file volume is made. The “instruction to reduce the filevolume” here may include, for example, an instruction to increaseavailable space of a recording medium for recording the image file, oran instruction to transfer data, which is more easily achieved when thefile volume is smaller.

When the instruction to reduce the file volume is made via the inputunit 10, the controller 12 starts the process, and the identificationcode is read from the header information contained in the image file F1(step ST21). Then, the header area and the image data area of the imagewhich has the identification code indicating that the image is theinterpolation image is deleted from the image file F1 (“deleteinterpolation image” in step ST22). Further, the header information ofeach of the remaining original images L1 and R1 is updated (step ST23).Specifically, the order of viewpoint in the header information of eachof the original images L1 and R1 is changed. That is, if the image fileF1 contains the original images L1 and R1 and the two interpolationimages H1 and H2, the values of the order of viewpoint of the originalimages L1 and R1 are 1 and 4, respectively, and after the interpolationimages are deleted, the values of the order of viewpoint of the originalimages L1 and R1 are changed to 1 and 2, respectively. Further,information about the interpolation images is deleted from the list ofthe address start positions contained in the header information of theoriginal image L1. Then, the media control unit 9 records the originalimage file F0 in the medium 9A (step ST24), and the process ends.

As described above, in this embodiment, the identification coderepresenting the type of the image, i.e., the original image or theinterpolation image, and the viewpoint information representing theviewpoint position are described in the header information of theoriginal images L1 and R1 and the interpolation image, and the originalimages L1 and R1 and the interpolation image each added with the headercontaining the identification code and the viewpoint information arestored as the single image file F1. This facilitates management of theoriginal images L1 and R1 and the interpolation image. In particular,since the identification code representing the type of the image isadded to each of the original images L1 and R1 and the interpolationimage, such a situation that the original image is erroneously deletedcan be prevented. Further, addition of the viewpoint informationrepresenting the viewpoint position facilitates generation of thestereoscopic image and re-creation of the interpolation image.

Furthermore, by storing the original images L1 and R1 and theinterpolation image as the single image file F1, the number of files isnot largely increased, thereby facilitating management of the originalimages L1 and R1 and the interpolation image.

Moreover, since the interpolation image is stored at the area followingthe areas for storing the original images L1 and R1 in the image fileF1, it is not necessary to re-create the file when the interpolationimage is deleted, thereby facilitating deletion of the interpolationimage.

In addition, when an instruction to reduce the file volume is made, onlythe interpolation image can be deleted, thereby facilitating reductionof the file volume.

Although the interpolation image is generated according to theinstruction to generate the interpolation image in the above-describedembodiment, the interpolation image may automatically be generated afterthe acquisition of the original images L1 and R1 through imaging.

In the above-described embodiment, the area of the interpolation imageis deleted when the instruction to reduce the file volume is made. Atthis time, the information of the interpolation number may be kept inthe header information to facilitate later recomposition of theinterpolation image from the original images L1 and R1.

Although the image processing device according to the invention isapplied to the stereoscopic imaging device including the imaging units2A and 2B in the above-described embodiment, the image processing deviceaccording to the invention may be provided alone. In this case, imagesacquired by imaging the same subject from different positions are fed tothe image processing device, and the image file F1 is generated in thesame manner as in the above-described embodiment. In this case, theimage file F1 can be generated in the same manner as in theabove-described embodiment not only when the interpolation image isgenerated from the original images L1 and R1 acquired through imaging,but also when, for example, the interpolation image is generated fromoriginal images generated using a computer graphics technique.

Although the original images L1 and R1 and the interpolation image areassociated to each other by storing the original images L1 and R1 andthe interpolation image in a single file in the above-describedembodiment, the original images L1 and R1 and the interpolation imagemay be associated to each other in any other manner as long as they areassociated in an integral fashion, such as storing the original imagesL1 and R1 and the interpolation image in the same folder, assigning acommon file name to the original images L1 and R1 and the interpolationimage, or recording, in the header of at least one of the originalimages L1 and R1 and the interpolation image, file names of the otherimages.

The image processing device 1 according to one embodiment of theinvention has been described. The invention may also be implemented inthe form of a program for causing a computer to function as meanscorresponding to the interpolation image generating unit 5 and the filegenerating unit 6 described above and to carry out the processes shownin FIGS. 9 and 10. The invention may also be implemented in the form ofa computer readable recording medium containing such a program.

What is claimed is:
 1. An image processing device, comprising: an imageacquiring unit for acquiring a plurality of original images of a subjectviewed from different viewpoints taken with a plurality of imagingunits; an interpolation image generating unit for determining whether aconvergence angle between the plurality of imaging units is less than apredetermined angle, and if the determination is negative, determining anumber of interpolation images to interpolate a new viewpoint betweenthe viewpoints of the original images, and generating, from the originalimages, the determined number of the interpolation images; and a storingunit for adding identification information and viewpoint information tothe original images and the interpolation image, the identificationinformation indicating whether each image comprises an original image ofthe original images or the interpolation image and the viewpointinformation representing an order of viewpoint of each image, andstoring the original images and the interpolation image in an associatedfashion with the identification information and the viewpointinformation added thereto.
 2. The image processing device as claimed inclaim 1, wherein the storing unit stores the original images and theinterpolation image as a single image file.
 3. The image processingdevice as claimed in claim 2, wherein the storing unit stores theinterpolation image at an area following areas for storing the originalimages in the image file.
 4. The image processing device as claimed inclaim 1, wherein, when an instruction to reduce a file volume is made,the storing unit deletes only the interpolation image.
 5. The imageprocessing device as claimed in claim 1, wherein the interpolation imagegenerating unit determines, as the number of the interpolation images, amaximum N value that provides a value less than the predetermined anglewhen the convergence angle is divided by (N+1).
 6. An imaging device,comprising: a plurality of imaging units disposed at different positionsfrom each other for acquiring a plurality of original images of asubject viewed from different viewpoints; and the image processingdevice as claimed in claim
 1. 7. An image processing method, comprising:acquiring a plurality of original images of a subject viewed fromdifferent viewpoints taken with a plurality of imaging units;determining whether a convergence angle between the plurality of imagingunits is less than a predetermined angle; if the determination isnegative, determining a number of interpolation images to interpolate anew viewpoint between the viewpoints of the original images; generating,from the original images, the determined number of the interpolationimages; and adding identification information and viewpoint informationto the original images and the interpolation image, the identificationinformation indicating whether each image comprises an original image ofthe original images or the interpolation image, and the viewpointinformation representing an order of viewpoint of each image, andstoring the original images and the interpolation image in an associatedfashion with the identification information and the viewpointinformation added thereto.
 8. A non-transitory computer readablerecording medium comprising a program for causing a computer to carryout an image processing method, the method comprising: acquiring aplurality of original images of a subject viewed from differentviewpoints taken with a plurality of imaging units; determining whethera convergence angle between the plurality of imaging units is less thana predetermined angle; if the determination is negative, determining anumber of interpolation images to interpolate a new viewpoint betweenthe viewpoints of the original images; generating, from the originalimages, the determined number of the interpolation images; and addingidentification information and viewpoint information to the originalimages and the interpolation image, the identification informationindicating whether each image comprises an original image of theoriginal images or the interpolation image, and the viewpointinformation representing an order of viewpoint of each image, andstoring the original images and the interpolation image in an associatedfashion with the identification information and the viewpointinformation added thereto.
 9. The image processing device as claimed inclaim 1, wherein the interpolation image generating unit determines theconvergence angle based on corresponding points between the originalimages.
 10. The image processing device as claimed in claim 1, whereinthe interpolation image generating unit determines whether informationof the convergence angle is added to the original images, and if thedetermination is affirmative, the interpolation image generating unitobtains the information of the convergence angle added to the originalimages, and if the determination is negative, the interpolation imagegenerating unit determines the convergence angle based on correspondingpoints between the original images.
 11. The image processing device asclaimed in claim 1, wherein the interpolation image generating unitestimates a viewpoint outside the viewpoints of the original images, andgenerates an interpolation image corresponding to the estimatedviewpoint.
 12. The image processing device as claimed in claim 1,wherein the interpolation image generating unit determines, as thenumber of the interpolation images, a maximum value that provides avalue less than the predetermined angle.
 13. The method of claim 7,wherein in said determining the number of the interpolation images, thenumber of the interpolation images is determined as a maximum N valuethat provides a value less than the predetermined angle when theconvergence angle is divided by (N+1).
 14. The method of claim 7,wherein in said determining the number of the interpolation images, thenumber of the interpolation images is determined as a maximum value thatprovides a value less than the predetermined angle.
 15. Thenon-transitory computer readable recording medium of claim 8, wherein insaid determining the number of the interpolation images, the number ofthe interpolation images is determined as a maximum N value thatprovides a value less than the predetermined angle when the convergenceangle is divided by (N+1).
 16. The non-transitory computer readablerecording medium of claim 8, wherein in said determining the number ofthe interpolation images, the number of the interpolation images isdetermined as a maximum value that provides a value less than thepredetermined angle.